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We determine the critical noise level for decoding low density parity check error correcting codes 
based on the magnetization enumerator (M), rather than on the weight enumerator (W) employed 
in the information theory literature. The interpretation of our method is appealingly simple, and 
the relation between the different decoding schemes such as typical pairs decoding, MAP, and finite 
temperature decoding (MPM) becomes clear. In addition, our analysis provides an explanation 
for the difference in performance between MN and Gallager codes. Our results are more optimistic 
than those derived via the methods of information theory and are in excellent agreement with recent 
results from another statistical physics approach. 
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I. INTRODUCTION 



The theory of error-correcting codes is based on the efficient introduction of redundancy to given messages for 
protecting the information content against corruption. The theoretical foundations of this area were laid by Shannon's 
seminal work |l| and have been developing ever since. One of the main results obtained in this field is the celebrated 
channel coding theorem stating that there exists a code such that the average message error probability Pe, when 
maximum likelihood decoding is used, can be made arbitrarily small for sufficiently long messages below the channel 
capacity; and will approach 1 above it. The channel coding theorem is based on unstructured random codes and 
impractical decoders such as maximum likelihood j| and typical set decoding j|. In the case of structured codes, 
the critical code rate R (message information content /length of the encoded transmission) may lie below the channel 
capacity, commonly termed Shannon's bound, even if optimal (and typically impractical) decoding methods are being 
used. The proximity of the critical code rate to Shannon's limit provides an indication to the theoretical limitations 
of a given code. 

In 1963 Gallager Q proposed a coding scheme which involves sparse linear transformations of binary messages that 
was forgotten soon after, in part due to the success of convolutional codes [|| and the computational limitations of 
the time. Gallager codes have been recently rediscovered by MacKay and Neal (MN), that independently proposed a 
closely related code ||. Variations of this family of codes, known as Low Density Parity Check (LDPC) codes, have 
displayed performance comparable (and sometimes superior) to other state-of-the-art codes. This family of codes has 
been thoroughly investigated in the information theory (IT) literature (e.g., |s| |^, |^|), providing a range of significant 
theoretical and practical results. 

In parallel to studies carried out in the IT community, a different approach has been used to study LDPC codes, 
using the established methods of statistical physics (SP). This analysis, relying mainly on the replica symmetric 
analysis of diluted systems jM |^| , offers an alternative to information theory methods and has yielded some new 
results and insights || [fl], 0|. Due to the growing interest in LDPC codes and their successful analysis via the 
methods of statistical physics, there is growing interest in the relationship between IT and SP methods. As the two 
communities investigate similar problems, one may expect that standard techniques known in one framework would 
bring about new developments in the other, and vice versa. Here we present a direct SP method to determine the 
critical noise level of Gallager and MN error correcting codes, which allows us to focus on the differences between the 
various decoding criteria and their use for defining the critical noise level for which decoding is theoretically feasible. 

The paper is organized as follows: In section II we introduce the general framework, notation and the quantities 
we focus on, while in section |l| we will briefly describe the SP calculation. Section |y| describes qualitatively the 
emerging picture of the main quantities calculated for Gallager's code while the corresponding picture for MN codes 
will be described in section |v|. Quantitative results for the critical noise level will be presented in section VI followed 
by conclusions. 
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II. REGULAR GALLAGER AND MN CODES 

In a general scenario, the N dimensional Boolean message s° £ {0,1}^ is encoded to the M(> N) dimensional 
Boolean vector t°, and transmitted via a noisy channel, which is taken here to be a Binary Symmetric Channel (BSC) 
characterized by an independent flip probability p per bit; other transmission channels may also be examined within 
a similar framework. At the other end of the channel, the corrupted codeword is decoded utilizing the structured 
codeword redundancy. 

The first type of error correcting code that we focus on here, is Gallager's linear code 0]. Gallager's code is a 
low density parity check code defined by the a binary (M-N)xM matrix A = [C1JC2], concatenating two very 
sparse matrices known to both sender and receiver, with the (M — N) x (M — N) matrix C2 being invertible. The 
matrix A has K non-zero elements per row and C per column, and the code rate is given by R—l — C/K—l — N/M. 
Encoding refers to multiplying the original message s° with the (MxN) matrix G T (where G = [HtvIC^ 1 ]), yielding 
the transmitted vector t°. Note that all operations are carried out in (mod 2) arithmetic. Upon sending t° through 
the binary symmetric channel (BSC) with noise level p, the vector r = t°+n° is received, where ri° is the true noise. 

Decoding is carried out by multiplying r by A to produce the syndrome vector z = Ar (= An", since AG T = 0). 
In order to reconstruct the original message s° , one has to obtain an estimate ft for the true noise n° . First we select 
all n that satisfy the parity checks An = An": 

Z pc (A, n°) = {n \ An = z}, and T pc (A, n°) = {ft £ Z pc (A, n°) \ n + n°}, (1) 

the (restricted) parity check set. 

The second type of error correcting code that we focus on here is the MacKay-Neal (MN) code [9 . An MN code 
is a low density parity check code defined by a binary Mx (N + M) matrix A = [C S |C„], concatenating two very 
sparse matrices known to both sender and receiver, with the M x M matrix C„ being invertible. The MxN matrix 
C s has K non-zero elements per row and C per column, while C„ has L non-zero elements per row and column. The 
code rate is given by i? = K/C = N/M . Encoding refers to multiplying the original message s° by the (MxN) dense 
generator matrix G = C~ 1 C S , yielding the transmitted vector t°. Note that all operations are carried out in (mod2) 
arithmetic. Upon sending t° through the binary symmetric channel (BSC) with noise level p, the vector r = t°+n° 
is received, where ri° is the true noise. 

Decoding is carried out by multiplying r by C n to produce the syndrome vector z — C s s°+C n n° = Ac°, where c is 
the concatenated vector (s, n). In order to reconstruct the original message s°, one has to obtain estimates c for the 
true signal and noise c°. First we select all combinations of signal and noise c that satisfy the parity checks Ac = Ac : 

2p C (A,c°) = {c| Ac = z}, and F pc (A,(?) = {c £ I pc (A,c°) \ (?}, (2) 

the (restricted) parity check set. 

To unify notation for Gallager and MN codes, we will adopt the notation c° for the original noise (and signal) 
vector, and c for the estimate of the noise (and signal) vector. Any general decoding scheme then consists of selecting 
a vector c* from X pc (A, c°), on the basis of some noise (and signal) statistics criterion. Upon successful decoding 
c° will be selected, while a decoding error is declared when a vector c* £ lp C (A, c°) is selected. For each decoding 
scheme, the average block error probability 

Pe(p s ,p) — (A ( a vector c £ 2p c (A, c°) is selected ) ) A ^ (3) 

can be defined as a measure of error correcting ability for a given code ensemble, where A(-) is an indicator function 
returning 1 if the proposition of the argument is true and 0, otherwise. For BSC, only the number of non-zero 
components characterizes the statistics of the noise. On the other hand, the signal bits in general have an equal 
probability for being and 1 (i.e. p s = i), which implies that they have no useful prior information for the estimation. 
In the following, we therefore focus on decoding schemes based on the weight of a vector which is the average sum of 
the noise components w(c) = ^2jLi n j- To obtain the error probability, one averages the indicator function over 
all c° vectors drawn from some distribution and the code ensemble A as denoted by (.) A 

Unfortunately, carrying out averages over the indicator function is difficult. Therefore, the error probability (^) is 
usually upper-bounded by averaging over the number of vectors n obeying a certain condition on the weight w(ri) 
which characterizes the employed decoding scheme. Alternatively, one can find the average number of vectors with a 
given weight value w from which one can construct a complete weight distribution of noise vectors n in Zp C (A, c°). 
From this distribution one can, in principle, calculate a bound for P e and derive critical noise values above which 
successful decoding cannot be carried out. 

A natural and direct measure for the average number of states is the entropy of a system under the restrictions 
described above, that can be calculated via the methods of statistical physics. 
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It was previously shown (see e.g. 0j for technical details) that this problem can be cast into a statistical mechanics 
formulation, by replacing the field ({0, 1}, +mod(2)) by ({1, —1}, x), and by adapting the parity checks correspond- 
ingly. The statistics of a noise vector n is now described by its magnetization m(n) = YljLi n ji ( m ("0 G [1> — 1]) ; 
which is inversely linked to the vector weight in the [0, 1] representation. Similarly, the statistics of a signal vector 



s is now described by its magnetization m s (s) = -^7X^=1 s j' ( m s(*) € [1 , — 1] ) . With this in mind, we introduce 
the conditioned magnetization enumerator, for a given code and noise, measuring the noise vector magnetization 
distribution in 2I C (A,7?°) 



To obtain the magnetization enumerator M.(m) 



M 



In 



Tr 5(m(n) — m) 



M(m) = ( M A , S o(m) ) 

\ / A,, 



(4) 



(5) 



which is the entropy of the noise vectors in Xp C (A, n°) with a given m, one carries out uniform explicit averages over 
all codes A with given parameters K, C (and L), and the weighted average over all possible noise vectors generated 
by the BSC, (and all possible signal vectors) i.e., 



A I 



P(n°) 



IJ((l-p) 5(n°-l)+p5(n° + l)), 
j 

N 



(6) 



(7) 



with p s = |. It is important to note that, in calculating the entropy, the average quantity of interest is the magne- 
tization enumerator rather than the actual number of states. As physicists, this is the natural way to carry out the 
averages for three main reasons: a) The entropy obtained in this way is believed to be self-averaging, i.e., its average 
value (over the disorder) coincides with its typical value, b) This quantity is extensive and grows linearly with the 
system size, c) This averaging distinguishes between annealed variables that are averaged or summed for a given set 
of quenched variables, that are averaged over later on. In this particular case, summation over all c vectors is carried 
for a fixed choice of code A and vector c°; averages over these variables are carried out at the next level. 

One should point out that in somewhat similar calculations, we showed that this method of carrying out the averages 
provides more accurate results in comparison to averaging over both sets of variables simultaneously . 

A positive magnetization enumerator, M(m)>0 indicates that there is an exponential number of solutions (in M) 
with magnetization m, for typically chosen A and c°, while A4(m)— >0 indicates that this number vanishes as M^oo 
(note that negative entropy is unphysical in discrete systems). 

Another important indicator for successful decoding is the overlap u> between the selected estimate ft* , and the true 

1 v^M 



u>(n, n°) 



A I 



Em 
.7 = 1 



(w(n,n°) G [—1, 1]), with u = 1 for successful (perfect) decoding. However, this 



quantity cannot be used for decoding as n° is unknown to the receiver. The (code and noise dependent) noise overlap 
enumerator is now defined as: 



WA^M^in 



Tr 6(u)(n,n°)—u>) 

ceXj c (A,c°) 



and the average quantity being 



W(w) = ( Wa,s>M) 



(8) 



(9) 



This measure is directly linked to the weight enumerator Q, although according to our notation, averages are car- 
ried out distinguishing between annealed and quenched variables unlike the common definition in the IT literature. 
However, as we will show below, the two types of averages provide identical results in this particular case. 
Similarly, for MN-codes one defines the signal magnetization and weight enumerators as 



M s (m s ) = i/ln 



W s (cj s ) ee - /In 



Tr 5(m(s)— ra,l 



Tr 5(u)(s, s°)— u s ) 
?gi; c (a,c°) 



(10) 
(11) 



A,c° 
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In what follows, we perform all calculations as if both m and ui (and m s and oj s for MN-codes), are constrained to 
particular values. As we will show, omitting a constraint in the final expressions can then easily be done by assigning 
the zero value to the corresponding Lagrange multiplier. 



III. THE STATISTICAL PHYSICS APPROACH 



Quantities of the type <2(c) = {Q y (c)) y , with Q y (c) = jj\n[Z y (c)] and Z y (c) = Tr x 5(c(x,y) — Mc), are very 
common in the SP of disordered systems; the macroscopic order parameter c(x, y) is fixed to a specific value and 
may depend both on the disorder y and on the microscopic variables x. Although we will not prove this here, such 
a quantity is generally believed to be self-averaging in the large system limit, i.e., obeying a probability distribution 
P {Qy{°)) = S(Q y (c) — Q(c))). The direct calculation of Q(c) is known as a quenched average over the disorder, but 
is typically hard to carry out and requires using the replica method ||] . The replica method makes use of the identity 
(In Z) — { lim n ^o['2 n ~l]/ n ), by calculating averages over a product of partition function replicas. Employing 
assumptions about replica symmetries and analytically continuing the variable n to zero, one obtains solutions which 
enable one to determine the state of the system. 

To simplify the calculation, one often employs the so-called annealed approximation, which consists of performing an 
average over Q y {c) first, followed by the logarithm operation. This avoids the replica method and provides (through 
the convexity of the logarithm function) an upper bound to the quenched quantity: 



(12) 



The technical details of the calculation are similar to those in |||. It turns out that it is useful to perform the 
gauge transformation Cj—>-CjC°, such that the averages over the code A and noise/signal c° can be separated, Wa,c° 
becomes independent of c°, leading to an equality between the quenched and annealed results, W(m) = A4 a (™)|p=o = 
Ai q {m)\ p= Q. For any finite noise value p one should multiply exp[W(w)] by the probability that a state obeys all parity 
checks exp[— K(u>,p)] given an overlap ui and a noise level p [||. In calculating W(lo) and A4 a / q (m), the (5-functions 
fixing m and u>, are enforced by introducing Lagrange multipliers m and Cj. 

Carrying out the averages explicitly one then employs the saddle point method to extremize the averaged quantity 
with respect to the parameters introduced while carrying out the calculation. These lead, in both quenched and 
annealed calculations, to a set of saddle point equations that are solved either analytically or numerically to obtain 
the final expression for the averaged quantity (entropy). 

The final expressions for the annealed entropy per noise degree of freedom for Gallager codes, under both overlap 
(w) and magnetization (to) constraints, are of the form: 



Q a = -§(ln(2) + (if-l)ln[l+cf])+ln/ Tr exp(7i(t2;+mn ))(l + ncf- 1 ) c '\ 

K \n=±i I n0 



— (ujco+rhrn) 



(13) 



where c\ has to be obtained from the saddle point equation ^2a. = 0. Similarly, the final expression in the quenched 
calculation, employing the simplest replica symmetry assumption m, is of the form: 



Qq = 



cjdxdx tt(x)tt(x) ln[l+xx] + — J | rfxfe7r(a;fc) j In 
c tt(x c ) I /in Tr^ exp(rt(o) + mn )) ]^[(1 




— (tDw+mm) 



(14) 



The probability distributions it(x) and n(x) emerge from the calculation; the former represents a probability distribu- 
tion with respect to the noise vector local magnetization [ flEfl , while the latter relates to a field of conjugate variables 
which emerge from the introduction of (5-functions while carrying out the averages (for details see §J). Their explicit 
forms are obtained from the functional saddle point equations j^jpj, j^fy — 0, and all integrals are from— 1 to 1. 

The final expressions for the annealed entropy per noise degree of freedom for MN-codes, under both signal and 
noise overlap (w,u> s ) and magnetization (to, m s ) constraints, are of the form: 



Qa 



(log(2) + (if+L-l)ln[l + cfdi]) - R(m s m s +LJ S LJ S ) - (mm+LJu;) 



+i?ln( Tr cxp (s(lj s + rh s s°)) (1 + sci) 

\ s=±l 



C 



hi 



Tr exp {n(Cj + m n°)) (1 + nd\) 

n— ±1 



(15) 
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where ci, di have to be obtained from the saddle point equations 7j§f-, = 0. Similarly, the final expression in the 
quenched calculation, employing the simplest replica symmetry assumption M, is of the form: 



(m s ms+0Js^s) — {mm+Luuj) 



k l r / k l \" 

Y[dx k Tt(x k ) Y[dyi p{yi) In J ( 1 + ^ ) ~ fi ( 

fc=i j=i L V fe=i ;=i /. 

K J dxdx it(x)tt(x) ln[l+xi] + R J dx c tt(x c ) ^ln 
L dydy p(y)p(y) ln[l+yy] + / \\dyi p(yi) /in Tr exp(n(u +mn°) JJ(1 



c 

Tr exp(s(w s + m s s°))TT(l 

S — ±1 ^ X 




(16) 



The probability distributions ir(x),p(y) and Tr(x),p(y) emerge from the calculation; the former represent probability 
distributions with respect to the signal/noise vector local magnetizations while the latter relate to fields of 
conjugate variables which emerge from the introduction of (5-functions while carrying out the averages (for details 
see ||). Their explicit forms are obtained from the functional saddle point equations j^jjpj, g^l) > s^y) ' Jffy) = ^' an< ^ 
all integrals are from— 1 to 1. 

Enforcing a 8- function corresponds to taking u),m,u) s ,rh a such that 



9Q a 



HQ., 



/l 



li 



dQ a 



0, while not 

enforcing it corresponds to putting to, to, cj s , rh s to 0. Since oj, to, uj Si m s , follow from 9 ^ a J q , -^f 2 -, d ^,~/ 9 , 9 ^2. /q = 0, 
all the relevant quantities can be recovered with appropriate choices of cD, to, <2> s , rh s . 



1 dm 1 duj s 5 drh s 



dm M 



a) p<p, 



M.(m) 



mo(p) 




c) p>p, 



m (p) 




m 1rQ.(p) 1 



FIG. 1: The qualitative picture of A4(m)>0 (solid lines) for different values of p. For MAP, MPM and typical set decoding, 
only the relative values of m+(p) and mo(p) determine the critical noise level. Dashed lines correspond to the energy contribution 
of — /3F at Nishimori's condition (/3 = 1). The states with the lowest free energy are indicated by a point •. a) Sub-critical noise 
levels p<p c , where m+(p) <mo(p), there are no solutions with higher magnetization than mo(j>), and the correct solution has 
the lowest free energy, b) Critical noise level p = p c , where m+(p) =mo(p). The minimum of the free energy of the sub-optimal 
solutions is equal to that of the correct solution at Nishimori's condition, c) Over-critical noise levels p > p c where many 
solutions have a higher magnetization than the true typical one. The minimum of the free energy of the sub-optimal solutions 
is lower than that of the correct solution. 
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IV. QUALITATIVE PICTURE 

We now discuss the qualitative behaviour of M{m) 1 and the interpretation of the various decoding schemes. To 
obtain separate results for M(m) and W(m) we calculate the results of Eqs.([l3|) and ( |l4| ) (and Eqs. ( pf ) and (p^|)), 
corresponding to the annealed and quenched cases respectively, setting Cj = to obtain M (m) and m = to obtain 
W(uj) (that becomes M.(m)\ p=0 after gauging). In Fig. |l[ we have qualitatively plotted the resulting function M,{m) 
for relevant values of p. M(m) (solid line) only takes positive values in the interval [m_(p), m+(p)]; for even K, M.(m) 
is an even function of m and m_(p) = — m+{p). The maximum value of A4(m) is always (1— R)hx{2) for Gallager 
codes, and i?ln(2) for MN codes. The true noise n° has (with probability 1) the typical magnetization of the BSC: 
m(n°) = mo(p) = l — 2p (dashed-dotted line). 

The various decoding schemes can be summarized as follows: 

• Maximum likelihood (MAP) decoding - minimizes the block error probability and consists of selecting 
the ft from T pc (A,n°) with the highest magnetization. Since the probability of error below m + (p) vanishes, 
P(3n G Xp C : m{n) > m + (p)) — 0, and since P(m(n°) =mo(p)) = 1, the critical noise level p c is determined by the 
condition m + (p c ) = mo(p c ). The selection process is explained in Fig.|l|(a)-(c). 

• Typical pairs decoding - is based on randomly selecting a n from 2" pc with m(n) — mo(p) ||; an error is 
declared when n° is not the only element of I pc . For the same reason as above, the critical noise level p c is 
determined by the condition m + (p c )=mQ{p c ). 

• Finite temperature (MPM) decoding - An energy -Fm(n) (with F=iln(±=£)) according to Nishimori's 
condition (corresponding to the selection of an accurate prior within the Bayesian framework) . is attributed to 
each ft 6 2p C , and a solution is chosen from those with the magnetization that minimizes the free energy J9|. This 
procedure is known to minimize the bit error probability fjjjfl . Using the thermodynamic relation T — IA — 4iS, 
(3 being the inverse temperature (Nishimori's condition corresponds to setting [3 = 1), the free energy of the 
sub-optimal solutions is given by T(m) = —Fm — ^M.{m) (for A4(m) > 0), while that of the correct solution 
is given by —Fmo(p) (its entropy being 0). The selection process is explained graphically in Fig.[j](a)-(c). The 
free energy differences between sub-optimal solutions relative to that of the correct solution in the current plots, 
are given by the orthogonal distance between A4(m) and the line with slope — f3F through the point (mo(p), 0). 
Solutions with a magnetization m for which M(m) lies above this line, have a lower free energy, while those 
for which M.(m) lies below, have a higher free energy Since negative entropy values are unphysical in discrete 
systems, only sub-optimal solutions with A4(m) > are considered. The lowest p value for which there are 
sub-optimal solutions with a free energy equal to —Fmo(p) is the critical noise level p c for MPM decoding. In 
fact, using the convexity of M(m) and Nishimori's condition, one can show that the slope dAi(m)/dm> —j3F 
for any value m<m (p) and any p, and equals — (3F only at m = m (p); therefore, the critical noise level for 
MPM decoding p=p c is identical to that of MAP, in agreement with results obtained in the information theory 
community Jl7| . 

The statistical physics interpretation of finite temperature decoding corresponds to making the specific choice for 
the Lagrange multiplier rh = (3F and considering the free energy instead of the entropy. In earlier work on MPM 
decoding in the SP framework |9| , negative entropy values were treated by adopting different replica symmetry 
assumptions, which effectively result in changing the inverse temperature, i.e., the Lagrange multiplier rh. This 
effectively sets m=m+-(p), i.e. to the highest value with non- negative entropy. The sub-optimal states with the 
lowest free energy are then those with m=m+.(p). 

The central point in all decoding schemes, is to select the correct solution only on the basis of its magnetization. 
As long as there are no sub-optimal solutions with the same magnetization, this is in principle possible. As shown 
here, all three decoding schemes discussed above, manage to do so. To find whether at a given p there exists a gap 
between the magnetization of the correct solution and that of the nearest sub-optimal solution, just requires plotting 
A4(m)(> 0) and mo(p), thus allowing a graphical determination of p c . Since MPM decoding is done at Nishimori's 
temperature, the simplest replica symmetry assumption is sufficient to describe the thermodynamically dominant 
state ||]. At p c the states with m+(p c ) —mo(p c ) are thermodynamically dominant, and the p c values that we obtain 
under this assumption are exact. 

V. MN CODES - AN ALTERNATIVE VIEW 

For MN codes there is a way to obtain the exact expression for A4, in the case of unbiased messages, by employing 
a single highly plausible assumption. We first note that every the parity check bit z <> = s° i ..s° K n° i ..n° L is made up 
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of a combination of K unbiased (i.e. p s = 5) signal bits, and L biased (i.e. p ^ |) noise bits. As a result, every 
syndrome element z<> is unbiased independently of the noise bit statistics. It is therefore plausible to assume that 
the noise bit statistics (i.e. p) have no influence on the distribution of the parity check bits 2<>, and therefore on M. 
(which only depends on the true noise through the z<>). If this assumption is satisfied, one can invoke Nishimori's 
condition to obtain an exact expression for M. 

Independently of the assumption, Nishimori's condition gives the following identity for the thermodynamically domi- 
nant state: 



dM{m) 



Since states characterized by any magnetization value m < mo (pt ) will become dominant for an appropriately chosen 
value of p, and since we assume that M. is independent of p, the identity 

««(»)! h (i±»\ _ (18) 



dm 2 \l—m 

must hold for any value of m. Furthermore, the maximum of _A/((m) is reached at m = with M.(0) = i?ln(2), and 
we have that 



M(m)=M(0) / du In [ — — ] = ln(2) 

2 Jo K^-uJ 



R-l + B 



1 + m 



(19) 



where i?2(p) is the binary entropy per bit for vectors with bias p. Hence, under this assumption, we do not only 
obtain the exact expression for yV((m), but we see that the critical noise level p c is given by R = 1 — H2(p c )i saturating 
Shannon's bound for this type of codes! 

Unfortunately, the assumption can not be verified easily without the replica method. To verify whether indeed 
d^Op"^ = 0; we have to take the derivative of expression ( |l6|) (setting & = u) s = m s = 0) with respect to p. It turns 
out that M. is only independent of p, when p(y) is an even function of y, which in turn requires that p(y) and tt(x) 
are even functions of their arguments. Numerical analysis shows, that this is the case for any K > 3 or K — 2, L > 3, 
while not so for K — 1 or K = L = 2. This result is consistent with those reported in i.e. that typical MN codes 
with K > 3 or K — 2, L > 3 do saturate Shannon's bound, while those with K = 1 and K = L = 2 do not. 

Intuitively this result can be understood in the following way. There are M parity check bits and only 7V(< M) 
signal bits, such that parity check bits, although individually unbiased, are not uncorrelated. These correlations do 
seem to have an effect on Ai(m) for K — 1 and K = L = 2, while for K > 3 and K = 2, L > 3 the signal bits seem to 
be "scrambled" enough in the parity checks for the correlations to be insignificant. Note that this argument does not 
hold for Gallager codes and MN codes with biased messages, where the parity check bits exclusively comprise biased 
bits, and are therefore biased themselves. They only become unbiased as K — > 00 for Gallager codes (for which it was 
already reported in the literature |5| that such codes can saturate Shannon's bound), and for K — ► 00 or L — > 00 for 
MN codes. 

In fact, numerical analysis reveals that for K > 3 and for K = 2, L > 3 we have that p(y) = 6(y), p(y) = S(y), 
tt(x) — S(x) at least up to m + (p) = mo(pt) which is independent of p. This allows us to calculate M. analytically 
from expression (|l^), and we recover, as expected, the exact expression (|I"9]). 

For K=l or K = L = 2, like in the case of Gallager codes, one can only obtain m+(p) numerically. The results 
of this procedure are presented in the next section. Furthermore, for K = 1 and for K = L = 2, we find that 
spontaneously m s ^ for some values of p < p c , when no restriction is enforced (i.e. for m s = 0). This implies that 
one may improve the decoding performance by imposing the condition of unbiased signal (similar to the conditions 
for typical set decoding), i.e. by adjusting the Lagrange multiplier rh s such that m 8 = 0. Unfortunately, this only 
happens for values of p for which there is an exponential number of sub-optimal solutions c£ Xp C (A, c°) with the same 
weight as c°, and imposing this constraint on the signal estimator only reduces this number, leaving it nevertheless, 
exponential. 

It was shown |^] that MN codes in principle contain sufficient information to saturate Shannon's bound for unbiased 
messages. For codes with K — I, or K = L = 2 7 some of this information is wasted in a region where errorless decoding 
is impossible anyway, such that Shannon's bound is not saturated. For codes with K > 3, or K = 2, L > 3, our 
analysis indicates that all information is used optimally, and that Shannon's bound can be theoretically saturated. 
Our argument also explains the relative importance of the parameters K and L for the behaviour of the code in 
comparison with C. 
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VI. CRITICAL NOISE LEVEL - RESULTS 



Some general comments can be made about the critical MAP (or typical set) values obtained via the annealed 
and quenched calculations. Since M q (m) < M. a {m) (for given values of K, C (L) and p), we can derive the general 
inequality p c „ >p c , a - For all K, C (L) values that we have numerically analyzed, for both annealed and quenched 
cases, Tn±(p) is a non increasing function of p, and p c is unique. The estimates of the critical noise levels p c , a /qi based 
on M. a i q , are obtained by numerically calculating m c a / q (p), and by determining their intersection with m (p). This 
is explained graphically in Fig.||(a). As the results for MPM decoding have already been presented elsewhere |0|, we 
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FIG. 2: a) Determining the critical noise levels p c , a /q based on the function M a / q for Gallager codes and for MN codes 
with K = l or K = L = 2, a qualitative picture, b) Comparison of different critical noise level (p c ) estimates for Gallager 
codes. Typical set decoding estimates have been obtained via the methods of IT Q, based on having a unique solution to 
W(m) = K(m,p c ), as well as using the methods of SP jL8|. The numerical precision is up to the last digit for the current 
method. Shannon's limit denotes the highest theoretically achievable critical noise level pt for any code ftl . 



will now concentrate on the critical results p c obtained for typical set and MAP decoding for Gallager codes; these 
are presented in Fig.|^(b), showing the values of p C}a i q for various choices of K and C compared with those reported 
in the literature. 

^,From the table it is clear that the annealed approximation gives a much more pessimistic estimate for p c . This is 
due to the fact that it overestimates M in the following way. M a (m) describes the combined entropy of n and n° as 
if ri° were thermal variables as well. Therefore, exponentially rare events for n° (i.e. m(n°) ^mg(p)) still may carry 
positive entropy due to the addition of a positive entropy term from n. In a separate study pq | these effects have been 
taken care of by the introduction of an extra exponent; this is not necessary in the current formalism as the quenched 
calculation automatically suppresses such contributions. The similarity between the results reported here and those 
obtained in is not surprising as the equations obtained in quenched calculations are similar to those obtained by 
averaging the upper-bound to the reliability exponent using a methods presented originally by Gallager [Q . Numerical 
differences between the two sets of results are probably due to the higher numerical precision here. 

We have also obtained the critical noise levels for some parameter choices in MN codes. We only present the 
quenched (exact) values, and compare them only with the highest theoretically achievable critical noise level pt for 
any code [jj], as we are not aware of values obtained with other methods in the literature. Note that although still 
strictly below pt , the critical noise levels p c for K = L = 2 with increasing values of C rapidly approach p t to within 
the current numerical precision. 
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0.15 


<0.174 


0.06 


0.174 
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0.174 


0.174 
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FIG. 3: a) Determining the critical noise levels p Cj9 based on the function A4 q for MN codes with K > 3 or K — 2, L > 3, 
a qualitative picture, b) Comparison of different critical noise level (p c , q ) estimates for MN codes. The numerical precision is 
up to the last digit for the current method. Shannon's limit denotes the highest theoretically achievable critical noise level pt 
for any code 0. 
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VII. CONCLUSIONS 



In this paper we have shown how both weight and magnetization enumerators can be calculated using the methods 
of statistical physics in the case of regular LDPC codes. We study the role played by the magnetization enumerator 
M.(m) in determining the achievable critical noise level for various decoding schemes. The formalism based on 
the magnetization enumerator A4 offers a intuitively simple alternative to the weight enumerator formalism used 
in conjunction with typical pairs decoding in the IT literature |}], ^8). The SP based analysis employes the replica 
method given the very low critical values obtained by the annealed approximation calculation. Furthermore, the 
powerfull gauge theory as proposed by Nishimori [8j , proves that the replica symmetric assumption is correct (at least 
at the critical noise level), and thus that the critical noise levels as obtained by our method are exact. Although 
we have concentrated here on the critical noise level for the BSC, other channel types as well as other quantities of 
interest can be treated using a similar formalism. The predictions for the critical noise level arc more optimistic than 
those reported in the IT literature, and are up to numerical precision in agreement with those reported in [ fl8| . We 
have also shown that the critical noise levels for typical pairs, MAP and MPM decoding must coincide, and we have 
provided an intuitive explanation to the difference between MAP and MPM decoding. Finally, an extension of this 
analysis to MN codes reveals the mechanism which allows them to saturate Shannon's limit for finite K > 3 and for 
K = 2, L > 3 values (if impractical algorithms such as maximum likelihood are used). This result, which is consistent 
with previous SP based analyses || is considered as surprising in the IT community. 

We believe that SP based analysis will provide more insight into the performance and characteristics of random 
LDPC codes, complementing the analysis provided by the methods of IT. 
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